Estimating the standard error of cross-Validation-Based estimators of classifier performance

First, we analyze the variance of Cross Validation (CV)-based estimators used for estimating performance classification rules. Second, propose a novel estimator to estimate this using Influence Function (IF) approach that had been previously very successfully bootstrap-based estimators. The motivation research is that, as best our knowledge, literature lacks rigorous method CV-based What available set ad-hoc procedures have no mathematical foundation since they ignore covariance structure among dependent random variables. conducted experiments show IF proposed has small RMS error with some bias. However, surprisingly, methods still work better than IF-based method. Unfortunately, due lack enough smoothness if compared bootstrap estimator. This opens three points: (1) more comprehensive simulation study clarify when wins or looses; (2) analysis figure out why well; and (3) treatment connection between appropriate amount “smoothness” decreasing bias